Technical Field
The present invention is generally directed to the application of a Discrete Recurrent Neural Network (“DRNN”) to solving signal processing tasks. More specifically, the present invention provides a method, implementable in hardware or software, for learning a DRNN to signal domains for the application of solving various signal processing tasks of practical value in ways that significantly further the art.
Description of the Related Art
Processing of natural signals is a crucial component of most technology products. For example, media companies, including but not limited to social networking websites like Facebook, are taxed with hosting many high-resolution images and videos on their servers. Better image and video compression algorithms allow for reduced storage space in the aforementioned servers.
Signal domains have become increasingly more complex, and at the same time larger amounts of data are being sent through the internet, local networks, etc. Thus, it is important to have methods for compressing relevant data in real-time. Moreover, learning structure of data in the desired signal processing domain can and should be exploited for significant practical benefit.
There are several commercially available software and hardware modules for the compression of natural signals. Typically, these existing modules use linear coding to decompose a particular signal (“Image”) into a weighted sum of basis functions, e.g. the discrete cosine transform (DCT) basis. The basis functions are sometimes chosen without learning a dictionary from the input data, and the continuous-valued weighting coefficients must be quantized.
Several methods exist for learning internal configurations (“Network Parameters”) in neural network approaches to signal processing, including supervised deep learning methods, which use back propagation for training on labeled data. Other methods use Bayesian ideas (e.g. Maximum Likelihood) or sampling (e.g. Contrastive Divergence). Typically, approaches suffer from lack of scalability to high dimensions and often do not learn structure from data in an unsupervised manner.
Those skilled in the art will appreciate the benefits of a convex objective function with an easily computed gradient for high-dimensional discrete DRNN estimation (DRNN “Training” or “Learning”). There are no local minima that hinder training or necessitate random restarts with such an objective function. The tractability of the gradient herein contrasts with more standard procedures, such as maximum likelihood estimation, in that the number of terms to evaluate grows linearly in the number of network nodes, rather than exponentially.
In view of the foregoing, the present invention fulfills technical requirements for a general system, needed in the art, which can efficiently learn structure in a given high-dimensional signal domain in an iterative online fashion (“Online Learning”), and in real-time efficiently utilize this learned structure for feature-extraction, compression, data similarity measurement, clustering and classification, error-correcting coding, among other signal processing tasks of significant practical value.